Introduction to GitHub, Git and Conda for Bioinformatics

Course objectives

At the end of this short course, you should understand:

What Git, GitHub and GitLab are.
Why software projects have dependencies.
Why bioinformatics pipelines can be difficult to install and reproduce.
What Conda environments are and how they solve many dependency problems.
The basic Conda commands.
How to install and run a bioinformatics project hosted on GitHub.

1. Git, GitHub and GitLab

1.1 What is Git?

Git is a version control system.

It keeps track of changes made to files over time.

Imagine that you are developing a bioinformatics pipeline:

pipeline.py
pipeline_v2.py
pipeline_v3.py
pipeline_final.py
pipeline_final_really_final.py
text

Instead of manually creating copies of your project, Git records the history:

A → B → C → D
text

Each point in the history is called a commit.

A commit is essentially a snapshot of the project at a particular moment.

Why is this useful?

You can:

see what changed :
- each commit comes with a message describing the changed
- you can compare files between different versions
know who changed it
go back to an older version
create experimental versions (branch)
work with several people
merge changes made by different developers

2. GitHub and GitLab

GitHub and GitLab are online platforms built around Git repositories.

A repository, or "repo", contains the files of a project together with its Git history.

A typical bioinformatics repository might look like:

my-pipeline/
├── README.md
├── scripts/
│   ├── preprocessing.py
│   └── analysis.R
├── config/
│   └── config.yaml
├── workflow/
│   └── workflow.smk
├── environment.yml
└── LICENSE
text

GitHub and GitLab add many services around Git:

online hosting;
collaboration;
issue tracking;
pull/merge requests;
code review;
documentation;
automated testing;
continuous integration;
release management.

3. Why Git/GitHub are useful for developers

Git is particularly useful for developers because it allows them to work safely on a project.

Version history

Suppose a developer changes 20 files and breaks the pipeline.

With Git, they can identify the problematic change and potentially return to a previous working version.

Branches

A developer can create a separate branch:

git checkout -b new-feature
bash

and work on a new feature without modifying the stable version.

The changes can later be merged into the main project.

Collaboration

Several developers can work on the same project.

For example:

Alice ── feature A ──┐
                     ├── main
Bob   ── feature B ──┘
text

Git helps combine their changes.

Code review

On GitHub/GitLab, developers can propose changes through a Pull Request (GitHub) or Merge Request (GitLab).

Other developers can review the code before it becomes part of the main project.

4. Why GitHub/GitLab are useful for end users

GitHub is not only useful for developers.

For a bioinformatics user, a repository can provide:

the source code;
documentation;
installation instructions;
example data;
configuration files;
previous releases;
bug/issues reports;

4.1 Open-source software

Many bioinformatics projects hosted on GitHub or GitLab are open source. This means that the source code is publicly available and can be inspected, modified and, depending on the project's license, redistributed.

For end users, this has several important advantages:

Understand how the software works: you can read the source code to see how an algorithm or pipeline is implemented, rather than treating the program as a "black box".
Inspect the code for suspicious behavior: being able to access the source code makes it possible to audit the software and look for potentially malicious or unexpected operations. However, open source does not automatically mean that software is safe — a proper security review may still be necessary.
Learn from existing projects: reading other people's code is an excellent way to learn programming, bioinformatics and good software-development practices.
Add new features: if the existing software does not do exactly what you need, you can modify it or develop an additional feature.
Contribute to the project: useful improvements can potentially be proposed to the original developers through a Pull Request.

For example, suppose you find a bioinformatics pipeline that performs 90% of the analysis you need. Because its source code is available, you can inspect how it works, modify it to add the missing functionality, and potentially contribute that improvement back to the community.

This is one of the fundamental ideas behind open-source scientific software:

       Existing project
              │
       ┌──────┴──────┐
       │             │
   Understand      Reuse
       │             │
       └──────┬──────┘
              │
          Modify / Improve
              │
              ▼
       New functionality
              │
              ▼
      Contribute back

Important: "open source" describes how the source code is made available; it is not a guarantee that the software is secure, correct, or scientifically valid. Users should still check the project's reputation, maintainers, activity, dependencies and license.

5. Software dependencies

A major problem in bioinformatics is that software rarely works completely by itself.

A program may depend on:

a specific version of Java;
a specific version of Python;
Python packages;
a specific version of R;
R packages;
system libraries;
external bioinformatics programs.

For example:

My pipeline
│
├── Python 3.10
│   ├── numpy
│   ├── pandas
│   └── pysam
│
├── R 4.3
│   ├── ggplot2
│   └── dplyr
│
├── Java 17
│
└── samtools
text

This collection of required software is called the project's dependencies.

6. Example: GATK

A real-world example is GATK from the Broad Institute.

The GATK repository is available on GitHub:

https://github.com/broadinstitute/gatk

GATK's current documentation lists several requirements, including:

Java   17
Python 3.10.13
R      4.3.1
text

and additional Python and R packages.

numpy  1.26.2
scipy  1.11.4
pandas 2.1.3
r-base 4.3.1
text

and other dependencies includes external bioinformatics tools such as:

samtools
bcftools
bedtools
tabix
text

with specific versions.

This is a perfect illustration of a dependency problem:

			                      GATK
				                   │
       ┌─────────────┼─────────────┼─────────────┼─────────────┼─────────────┐
       │             │             │             │             │             │
    Java 17      Python 3.10      R 4.3       samtools      bcftools      bedtools     
       │             │             │             │             │             │
text

8. Why not simply install everything on the server?

Imagine that we have one shared server.

For Pipeline A, we install:

Python 3.12
Java 21
R 4.5
samtools 1.22
text

Everything is fine... Until we try to run an older Pipeline B, that requires:

Python 3.8
Java 11
R 4.1
samtools 1.15
text

Now we have a problem.

8.1 Version conflicts

Some programs require different versions of the same software.

For example:

Pipeline A → Python 3.12
Pipeline B → Python 3.8
text

Installing one version globally may break the other pipeline.

The same problem can occur with:

Java
R
samtools
numpy
pandas
Bioconductor packages
system libraries
text

8.2. The "one server" problem

Suppose we have:

                      SERVER
                        │
        ┌───────────────┼───────────────┐
        │               │               │
    Pipeline A      Pipeline B      Pipeline C
        │               │               │
    Python 3.8      Python 3.10     Python 3.12
    Java   11       Java   17       Java   21
    R      4.1      R      4.3      R      4.5
text

A traditional solution would be to install everything globally.

But then we can get:

Pipeline A breaks
       ↓
upgrade Python
       ↓
Pipeline B breaks
       ↓
upgrade R
       ↓
Pipeline C breaks
text

This is sometimes called dependency hell.

9. What is Conda?

Conda is a package manager and environment manager.

Instead of installing software globally on the computer, Conda can create separate environments.

For example:

Computer
│
├── Environment: pipeline_A
│   ├── Python 3.8
│   ├── R 4.1
│   └── samtools 1.15
│
├── Environment: pipeline_B
│   ├── Python 3.10
│   ├── R 4.3
│   └── samtools 1.18
│
└── Environment: pipeline_C
    ├── Python 3.12
    ├── R 4.5
    └── samtools 1.22
text

The environments are largely independent.

This means that different projects can use different software versions on the same computer.

10. The key idea behind Conda

Think of a Conda environment as a box containing the software required by one project.

For example:

┌──────────────────────────────┐
│     RNA-seq environment      │
│                              │
│ Python 3.10                  │
│ pandas 2.x                   │
│ R 4.3                        │
│ samtools 1.x                 │
│ fastqc                       │
│ multiqc                      │
└──────────────────────────────┘
text

Another project can have its own box:

┌──────────────────────────────┐
│     Variant environment      │
│                              │
│ Python 3.8                   │
│ Java 17                      │
│ samtools                     │
│ bcftools                     │
│ GATK                         │
└──────────────────────────────┘
text

The two environments can coexist.

11. Conda environments

Create an environment

conda create -n myproject
bash

Here:

create = create an environment
-n = name
myproject = environment name

Create an environment with a specific Python version

conda create -n myproject python=3.10
bash

Activate (enter) an environment

conda activate myproject
bash

After activation, commands use the software installed in that environment. Software installed globally on the server are still available if not overridden by the environment.

Deactivate (exit) an environment

conda deactivate
bash

List environments

conda env list
bash

or:

conda info --envs
bash

Install a package

Inside an activated environment:

conda install numpy
bash

For example:

conda install samtools
bash

Install a specific version

conda install samtools=1.20
bash

Similarly:

conda install python=3.10
bash

List installed packages

conda list
bash

Remove an environment

conda env remove -n myproject
bash

12. Creating an environment from a YAML file

This is one of the most important features for bioinformatics.

A project can provide a file such as:

environment.yml
text

For example:

name: myproject

channels:
  - conda-forge
  - bioconda

dependencies:
  - python=3.10
  - pandas
  - samtools
  - fastqc
yaml

Instead of manually installing everything, we can run:

conda env create -f environment.yml
bash

Conda reads the file and attempts to construct the required environment.

This is much more reproducible than giving users a list such as:

Install Python.
Then install pandas.
Then install samtools.
Then install...
text

13. The complete picture

We can now connect GitHub and Conda.

A well-organized bioinformatics project might look like:

GitHub repository
│
├── README.md
├── scripts/
├── workflow/
├── config/
└── environment.yml
text

The README explains:

How to install the project
How to create the environment
How to run the pipeline
text

The environment.yml describes:

Python version
Python packages
R version
R packages
samtools
other dependencies
text

The user can therefore go from:

GitHub repository
       ↓
clone repository
       ↓
create Conda environment
       ↓
activate environment
       ↓
run pipeline
text

14. Installing a GitHub bioinformatics project

Let's imagine that we want to install a project called:

my-bioinformatics-project
text

hosted on GitHub.

Step 1 — Install Git

Check whether Git is available:

git --version
bash

If it is installed, you should get something similar to:

git version 2.34.1
text

Step 2 — Clone the repository

Find the repository URL on GitHub.

For example:

git clone https://github.com/user/my-bioinformatics-project.git
bash

This downloads the repository to your computer.

Then:

cd my-bioinformatics-project
bash

You are now inside the project directory.

Step 3 — Read the README

Do not immediately start installing random dependencies.

First read:

README.md
text

Look for sections such as:

Installation
Requirements
Dependencies
Usage
Quick start
Conda
text

The README is usually the first source of information about how the developers expect the software to be installed.

Step 4 — Look for an environment file

Check the repository:

ls
bash

Look for files such as:

environment.yml
environment.yaml
conda.yml
text

You might also find:

requirements.txt
pyproject.toml
renv.lock
Dockerfile
text

These files describe different aspects of the project's software environment.

Step 5 — Create the Conda environment

If the project provides:

environment.yml
text

run:

conda env create -f environment.yml
bash

Conda will read the file and install the specified dependencies.

Step 6 — Activate the environment

The environment name is usually specified near the beginning of the YAML file:

name: myproject
yaml

Activate it:

conda activate myproject
bash

Check the environment:

conda env list
bash

You should see something similar to:

base
myproject   *
text

The * indicates the currently active environment.

Step 7 — Check the important programs

For example:

python --version
bash

R --version
bash

java -version
bash

samtools --version
bash

The versions should correspond to what the project expects.

Step 8 — Run the example

The README will normally provide an example.

For example:

python scripts/analyze.py \
    --input data/example.bam \
    --output results/
bash

Or perhaps:

snakemake --cores 4
bash

The exact command depends on the project.

15. Why this is useful for reproducible research

Suppose you publish a bioinformatics analysis.

You provide:

GitHub repository
        +
environment.yml
        +
documentation
        +
analysis code
text

Another researcher can obtain the code and reconstruct an environment containing the required software versions even years later.

This does not guarantee perfect reproducibility in every situation — operating systems, hardware, external databases, random seeds, input data and other factors can also matter — but it is a major improvement over simply saying:

"Install Python and run this script."

16. Cheat sheet

Git

For users :

git clone <URL> # download a repository
git pull        # download new changes
bash

For developers/contributors

git status          # see changes
git add <file>      # select changes
git commit -m "..." # save a snapshot
git push            # upload your changes
bash

Conda

conda create -n myenv python=3.10   # Create a new env, with a given version of python
conda env create -f environment.yml # Create a new env from a YML file
conda activate myenv                # Activate (enter) an environment
conda deactivate                    # Deactivate (exit) current enviromnent

conda install samtools              # Install a software
conda install python=3.10           # Install a software in a particular version

conda env list                      # List existing environments
conda list                          # List packages installed in current environment

conda env remove -n myenv           # Remove (destroy) existing environment
bash

17. Take-home messages

Git

Git tracks changes to files.

GitHub / GitLab

GitHub and GitLab host Git repositories and provide collaboration tools.

Dependencies

Software often depends on other software and on specific versions.

The server problem

Different projects may require incompatible versions of the same software.

Conda

Conda creates isolated environments containing the software versions required by different projects.

Environment file

An environment.yml file describes how to recreate a project's Conda environment.

Reproducibility

Code + documented dependencies + environment specification makes it much easier for someone else to reproduce an analysis.

18. Concrete examples

18.1. CheckM2

Clone the project from GitHub and enter the directory

cd
git clone --recursive https://github.com/chklovski/checkm2.git
cd checkm2
bash

If needed, deactivate the current environment

conda deactivate
bash

If needed, accept the terms and conditions

conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main
conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r
bash

Create the environment from the YML description

conda env create -n checkm2 -f checkm2.yml
bash

Activate the environment

conda activate checkm2
bash

Check that the python version you are using comes indeed from your conda env

which python
bash

Launch the install

python setup.py install
bash

Test the program

bin/checkm2 -h
bash

18.2. BacPP: BacterialPloidyPredictor

If needed, deactivate the current environment

cd
conda deactivate
bash

Clone the project from GitHub and enter the directory

git clone https://github.com/Dias-Lab/BacterialPloidyPredictor-v1.git
cd BacterialPloidyPredictor-v1/
bash

Create the environment from the YML description

conda env create -n bacpp -f bacpp.yml
bash

Activate the environment

conda activate bacpp
bash

Test the program

./bacpp.py -h
bash

18.3. VCFProcessor

If needed, deactivate the current environment

cd
conda deactivate
bash

Create an environment containing java17

conda create -n java17
bash

Activate the environment

conda activate java17
bash

Install openjdk 17

conda install openjdk=17
bash

Download VCFProcessor

wget https://github.com/ThomasLudwig/VCFProcessor/releases/download/1.2.13/VCFProcessor-1.2.13.jar
bash

Test the application

java -jar VCFProcessor-1.2.13.jar
bash